Skip to content

feat: GoThirdPartyCombinedLoader — CDN-first + local fallback routing#648

Merged
shivasurya merged 11 commits intomainfrom
shiva/pr-08-golang-thirdparty
Apr 7, 2026
Merged

feat: GoThirdPartyCombinedLoader — CDN-first + local fallback routing#648
shivasurya merged 11 commits intomainfrom
shiva/pr-08-golang-thirdparty

Conversation

@shivasurya
Copy link
Copy Markdown
Owner

Summary

  • Add GoThirdPartyCombinedLoader wrapping CDN and local GoThirdPartyLoader instances with CDN-first resolution
  • CDN transient errors (non-nil err) fall through to local; authoritative miss (nil, nil) stops resolution to avoid masking CDN data bugs
  • InitGoThirdPartyLoader creates a combined loader when CDN manifest loads; falls back to local-only on CDN failure or absent logger
  • CPF_CDN_URL env var overrides the CDN base URL at runtime; package-level thirdPartyRegistryBaseURL is test-overridable

Test plan

  • go test ./graph/callgraph/registry/ ./graph/callgraph/builder/ -count=1 — all pass
  • golangci-lint run ./graph/callgraph/... ./tools/... — 0 issues
  • CDN-first resolution: authoritative miss does not fall back to local
  • Transient CDN error falls back to local loader transparently
  • nil CDN or local loader degrades gracefully (skipped silently)

🤖 Generated with Claude Code

@shivasurya shivasurya added enhancement New feature or request go Pull requests that update go code labels Apr 5, 2026
@shivasurya shivasurya self-assigned this Apr 5, 2026
@shivasurya shivasurya added enhancement New feature or request go Pull requests that update go code labels Apr 5, 2026
@safedep
Copy link
Copy Markdown

safedep bot commented Apr 5, 2026

SafeDep Report Summary

Green Malicious Packages Badge Green Vulnerable Packages Badge Green Risky License Badge

Package Details
Package Malware Vulnerability Risky License Report
icon golang.org/x/mod @ v0.34.0
sast-engine/tools/validate_go_resolution/go.mod
ok icon
ok icon
ok icon
🔗
icon golang.org/x/sync @ v0.20.0
sast-engine/tools/validate_go_resolution/go.mod
ok icon
ok icon
ok icon
🔗
icon golang.org/x/tools @ v0.43.0
sast-engine/tools/validate_go_resolution/go.mod
ok icon
ok icon
ok icon
🔗

View complete scan results →

This report is generated by SafeDep Github App

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 5, 2026

Code Pathfinder Security Scan

Pass Critical High Medium Low Info

No security issues detected.

Metric Value
Files Scanned 46
Rules 205

Powered by Code Pathfinder

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 5, 2026

Codecov Report

❌ Patch coverage is 85.71429% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.59%. Comparing base (e51b501) to head (da0ff4e).

Files with missing lines Patch % Lines
sast-engine/cmd/resolution_report.go 8.33% 32 Missing and 1 partial ⚠️
sast-engine/graph/callgraph/builder/helpers.go 50.00% 12 Missing and 2 partials ⚠️
...-engine/graph/callgraph/extraction/go_variables.go 87.62% 9 Missing and 3 partials ⚠️
sast-engine/graph/callgraph/builder/go_version.go 86.20% 3 Missing and 1 partial ⚠️
sast-engine/cmd/ci.go 0.00% 1 Missing ⚠️
sast-engine/cmd/scan.go 0.00% 1 Missing ⚠️
sast-engine/cmd/serve.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@                        Coverage Diff                        @@
##           shiva/pr-07-golang-thirdparty     #648      +/-   ##
=================================================================
+ Coverage                          84.45%   84.59%   +0.13%     
=================================================================
  Files                                163      164       +1     
  Lines                              23594    23915     +321     
=================================================================
+ Hits                               19926    20230     +304     
- Misses                              2920     2937      +17     
  Partials                             748      748              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Owner Author

shivasurya commented Apr 7, 2026

Merge activity

  • Apr 7, 8:35 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Apr 7, 8:49 PM UTC: Graphite rebased this pull request as part of a merge.
  • Apr 7, 8:49 PM UTC: @shivasurya merged this pull request with Graphite.

@shivasurya shivasurya changed the base branch from shiva/pr-07-golang-thirdparty to graphite-base/648 April 7, 2026 20:46
@shivasurya shivasurya changed the base branch from graphite-base/648 to main April 7, 2026 20:47
shivasurya and others added 10 commits April 7, 2026 20:48
…ting

- Add GoThirdPartyCombinedLoader wrapping CDN and local GoThirdPartyLoader
  instances with CDN-first resolution and authoritative-miss semantics
- CDN transient errors (non-nil err) fall through to local; authoritative
  miss (nil, nil) stops resolution to avoid masking CDN extraction bugs
- InitGoThirdPartyLoader creates combined loader when CDN manifest loads
  successfully; falls back to local-only on CDN failure or absent logger
- CPF_CDN_URL env var overrides the CDN base URL at runtime

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion

Three targeted fixes to the Go call graph builder hot path:

1. buildParentMap once: findContainingGoFunction and findParentGoFunction
   previously rebuilt the full child→parent reverse-edge map on every call
   node (225k nodes × 65k calls ≈ 14.7B iterations on Ollama). Both now
   accept a pre-built parentMap passed from their call sites.

2. buildPkgVarIndex once: Source 3 in resolveGoCallTarget previously did a
   full O(N) scan of all CodeGraph nodes per unresolved method call to find
   package-level variables. Replaced with a map[dir::varName]*Node index
   built once before Pass 4, making each lookup O(1).

3. isBuiltin package-level var: promoted the builtin name map from a
   per-call allocation to a package-level var.

All existing tests pass unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- resolveGoCallTarget now returns resolveSource as 4th value:
  "thirdparty_local" for Pattern 1a import calls and Check 2.5 method
  calls into third-party packages; "" for all other paths
- Outer resolution loop propagates resolveSource to CallSite.TypeSource
  so resolution-report stats correctly count third-party resolutions
  instead of mislabeling them as user-code
- Remove unused codeGraph parameter from resolveGoCallTarget
- Add built-in type names (int, float64, string, rune, byte, etc.) to
  goBuiltins so T(x) type-conversion expressions resolve to
  builtin.T instead of being counted as unresolved function calls
- Add TestThirdPartyResolution_TypeSourceLabeling verifying both
  Pattern 1a and Check 2.5 paths emit TypeSource=thirdparty_local

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three new call resolution improvements:

1. Receiver variable binding (go_variables.go):
   - Extracts receiver name (e.g. `s` in `func (s *Store) Method()`)
   - Adds typed GoVariableBinding to the method scope
   - Fixes `s.db.Query()`, `s.sched.Run()` etc. inside method bodies
   - New helper: extractReceiverName()

2. var_declaration support (go_variables.go):
   - Handles `var sb strings.Builder`, `var mu sync.Mutex` etc.
   - Uses explicit type annotation (confidence 0.9) when present
   - Falls back to RHS inference for `var x = someFunc()`
   - New functions: processVarDeclaration(), processVarSpec()

3. Struct field index + Source 4 (go_builder.go, core/types.go):
   - Builds GoStructFieldIndex: "PkgType.Field" → resolved field type FQN
   - Source 4 splits ObjectName="a.KNorm" → looks up root type then field
   - Resolves `a.KNorm.Forward()`, `m.TextEncoder.Encode()`, `db.conn.QueryRow()`
   - No signature changes: index stored in CallGraph.GoStructFieldIndex

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The two [debug-1b] Fprintf(os.Stderr) calls in resolveGoCallTarget fired
unconditionally for every method call where scope lookup missed — tens of
thousands of syscalls on large projects like Ollama.

- Add logger *output.Logger parameter to BuildGoCallGraph and
  resolveGoCallTarget (nil-safe: tests pass nil, cmd callers pass logger)
- Replace both fmt.Fprintf calls with logger.Debug(), guarded by
  logger.IsDebug() — only fires when --debug / VerbosityDebug is active
- All four cmd entry points (scan, ci, serve, resolution-report) thread
  the existing logger through

All tests pass unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t field index

- go_variables_vardecl_test.go: 11 tests covering processVarDeclaration /
  processVarSpec (stdlib qualified, bytes.Buffer, sync.Mutex, net/url alias,
  multi-name, grouped var block, unqualified same-pkg, method body scope,
  RHS value fallback) and extractReceiverName (pointer + value receivers)
- go_builder_structfield_test.go: 6 tests covering buildStructFieldIndex
  (basic mapping, same-package field, non-struct skip, embedded field skip)
  and Source 4 end-to-end (Attention.KNorm.Forward, Store.db.QueryRow)

Also fixes processVarSpec condition: typeFQN != typeStr was incorrectly
preventing bindings for stdlib packages where alias == import path
(e.g. strings→strings, bytes→bytes).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ply pattern

Split the sequential Pass 4 resolution loop into two stages:
- Stage 1: N workers each resolve a shard of callsites independently,
  collecting (callerFQN, targetFQN, callSite) results into per-shard
  local slices. All reads in resolveGoCallTarget are immutable-by-Pass-4
  or RWMutex-protected; no shared writes occur during this stage.
- Stage 2: Single goroutine applies shard results sequentially via
  AddEdge/AddCallSite, eliminating the need for any mutex on the
  call graph write path.

Progress is tracked via atomic.Int64 with \r overwrite matching
the existing Pass 2b display pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add go_builder_pass4_test.go with 23 targeted tests covering all
previously uncovered branches in the parallel Pass 4 code and the
helper functions it depends on:

Pass 4 worker body:
- TestPass4_UnresolvedCallSiteRecorded  — unresolved else branch (lines 289-303)
- TestPass4_Source2EnrichmentStripsPointerPrefix — * stripping in Source 2 enrichment (lines 251-253)

Debug logger paths:
- TestResolveGoCallTarget_DebugLoggerScopeNoBinding (lines 587-589)
- TestResolveGoCallTarget_DebugLoggerNoScope (lines 590-592)

S4 struct field chained resolution:
- TestResolveGoCallTarget_S4Source1FunctionParam (lines 624-630)
- TestResolveGoCallTarget_S4Source3PkgVar (lines 644-648)

Stdlib/ThirdParty loader paths:
- TestResolveGoCallTarget_StdlibLoaderMethodFound (lines 674-676)
- TestResolveGoCallTarget_ThirdPartyLoaderFound (lines 669-676)
- TestResolveGoCallTarget_PromotedMethodViaCheck3 (lines 702-704)

Pattern 4 unresolved:
- TestResolveGoCallTarget_Pattern4Unresolved (line 740)

buildStructFieldIndex edge cases:
- TestBuildStructFieldIndex_DirNotInRegistry (lines 849-850)
- TestBuildStructFieldIndex_EmptyTypeAfterPointerStrip (lines 863-864)

Parent-map multilevel walk:
- TestFindContainingGoFunction_MultilevelWalk (line 901)
- TestFindParentGoFunction_MultilevelWalk (line 923)

resolvePromotedMethod with StdlibLoader (lines 1040-1050):
- TestResolvePromotedMethod_StdlibLoaderTypeNotFound
- TestResolvePromotedMethod_StdlibLoaderInvalidFQN
- TestResolvePromotedMethod_StdlibLoaderCallsFromFields

resolvePromotedMethodFromFields method found (lines 1071-1076):
- TestResolvePromotedMethodFromFields_MethodFoundInEmbedded

All 29 packages pass; builder coverage 80.9% → 83.2%.
Remaining 3 uncovered ranges are pre-existing progress-print paths
requiring 5000+ nodes and an import-extraction error path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…olution

- Add GetPackage to GoStdlibLoader interface for package-wide type scanning
- Check 2b: scan package interfaces for promoted methods not listed on the
  concrete type in CDN data (e.g. testing.T.Fatalf → testing.TB.Fatalf)
- S4-Source4b: lazy CDN lookup for stdlib struct fields (net/http.Request.Header
  → net/http.Header) when user-code struct index has no entry
- Expand short-qualified CDN field types (url.URL) using calling file's
  importMap (url → net/url) before Check 4 rejects them as incomplete FQNs
- Remove GoStructFieldIndex guard on Source 4 so S4-Source4b fires even for
  projects with no user-defined struct fields
- Add gopls-based ground-truth validation tool under tools/validate_go_resolution
- Improve Go call resolution: 79.1% → 88.4% on ollama, 94.8% on sast-engine

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ion_report

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix summaryConfirmsFlow to accept ParamToSink as flow confirmation,
  enabling detection from HTTP handler functions (void, no return taint)
  that call a source internally and pass taint to callees.
- Fix global dedup to key by sink only (not source+sink), preventing
  the same sink from appearing once per source handler.
- Add SourceSnippet/SourceLocation to EnrichedDetection; text output
  now shows Source: <file:line> + snippet and Sink: <file:line> + snippet
  for Taint-Global findings.
- Rewrite command_injection.py and sql_injection.py as L1 QueryType rules;
  remove L4 path_traversal.py (bare calls, no source constraint).
- Add OLLAMA-SEC-001/002/003 rules targeting exec, filepath, and outbound
  HTTP sinks from gin/net-http sources.
- Add security_flows test fixture for verifying global inter-procedural
  taint detection end-to-end.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@shivasurya shivasurya force-pushed the shiva/pr-08-golang-thirdparty branch from 197ba23 to 1e6c614 Compare April 7, 2026 20:48
@shivasurya shivasurya merged commit 00edcd8 into main Apr 7, 2026
5 of 6 checks passed
@shivasurya shivasurya deleted the shiva/pr-08-golang-thirdparty branch April 7, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request go Pull requests that update go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant